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Abstract 


This study examines the extent to which conceptual change is stimulated and 
achieved through online discussion in the context of an online graduate course. 
Transcripts of discussions among 15 graduate students studying assessment issues 
in mathematics and science education were analyzed using an interaction analysis 
model developed to assess knowledge construction and conceptual change in the 
context of an online debate. Although evidence of authentic conceptual change 
was limited, significant cognitive activity supporting conceptual change was iden- 
tified. We offer observations about course characteristics that contributed to con- 
ceptual change, discuss the limitations of applying an existing content analysis 
model to a new context, and suggest further research focusing on mathematical 
discourse and scientific inquiry in the online environment. 

Resume 


Cette etude examine jusqu'a quel point un changement conceptuel est stimule et 
realise a travers une discussion en ligne dans le contexte d'un cours de deuxieme 
cycle en ligne. Les transcriptions des discussions entre quinze etudiants gradues 
etudiant revaluation en mathematique et en sciences de T education ont ete analy- 
sees grace a un modele d'analyse de Tinteraction developpe pour mesurer la 
construction de connaissance et le changement conceptuel dans le contexte d'un 
debat en ligne. Malgre le fait que les preuves qu'un changement conceptuel au- 
thentique se soit produit sont limitees, une activite cognitive significative suppor- 
tant le changement conceptuel a ete identifiee. Des observations sont presentees 
sur les caracteristiques d'un cours contribuant au changement conceptuel, de 
meme qu'une discussion sur les limites de l'application d'un modele d'analyse de 
contenu dans un nouveau contexte. Des suggestions sont faites pour de nouvelles 
etudes centrees sur le discours en mathematique et en recherche scientifique dans 
les environnements en ligne. 


Introduction 

Distance learning modes, particularly computer-mediated communica- 
tion (CMC), have opened new pathways to higher-level thinking via dis- 
cussion, reflection, and negotiated meaning. CMC has also enhanced the 
ability to study not only the outcomes of interaction, but the process of 
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interaction itself as the vehicle by which larger purposes are achieved 
(Wells, 1999). Analysis of discourse — in this case, online discussion — lends 
itself not only to assessing whether an individual has achieved the end of 
acquiring new knowledge, but provides an avenue for studying the pro- 
cess of acquiring that knowledge — the means by which new knowledge is 
achieved — as a phenomenon in itself. This article investigates the potential 
for promoting and supporting conceptual change via online discussion in 
the context of graduate-level coursework among mathematics and science 
educators. 

The proliferation of various forms of online interaction in educational 
settings not only brings a new perspective to the study of conceptual 
change and the process of knowledge construction, but has augmented the 
techniques available for data collection and analysis in this arena. CMC 
supplies a wealth of data in the form of detailed, electronically archived 
transcripts that allow researchers to "observe" online interaction in the 
same environment in which participants experienced it. The flexible 
structure of online discussion also supports rich and diverse forms of 
analysis: for example, a variety of topical threads can be followed in a 
single discussion; several discussions can be compared; or a series of 
discussions can be analyzed to track changes over time. 

The effectiveness of CMC as an educational tool has received attention 
in research in terms of quantifiable outcomes such as patterns of interac- 
tion and degree of participant satisfaction (Fahy, 2003; Kanuka & Ander- 
son, 1998; Stacey, 1999). However, answers to questions about the quality, 
content, and cognitive level of online discussion are more elusive. In an 
extensive review of research on distance learning in education settings in 
the 1990s, Phipps and Merisotis (1999) identified only 40 studies based on 
original research (dedicated to explaining or predicting phenomena re- 
lated to distance learning) from among several hundred articles examined. 
In their content analysis of the Journal of Distance Education from 1986 
through 2001, Rourke and Szabo (2002) found that most published articles 
were descriptive in nature, with a very small percentage based on theoreti- 
cal research. The results of this study add to the empirical research base 
regarding quality of discussion and evidence of conceptual change in 
online discourse. 


Purpose 

This study addresses the question. To what extent is there evidence of 
conceptual change through online discussion among graduate students in 
a CMC course environment? Of related interest is whether the course 
structure provides adequate scaffolding to nurture conceptual change. To 
explore these questions, we searched for evidence of conceptual change 
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among students studying assessment issues in mathematics and science 
education in a fully asynchronous online graduate course. We applied 
methods of content analysis to individual messages, but expanded this 
with a more global analysis of overall context and the cumulative effect of 
message sequences. 

In their work on critical thinking. Garrison, Anderson, and Archer 
(2001) identify cognitive activity as "both a process and an outcome.. . . the 
acquisition of deep and meaningful understanding as well as content- 
specific critical inquiry abilities, skills, and dispositions" (p. 8). In a similar 
manner, we interpret conceptual change as both an outcome (evidenced 
by deeper understanding and new knowledge of the topic at hand — in this 
case, assessment) and as a process (evidenced by increasingly sophisti- 
cated levels of cognitive activity, efforts at synthesis and analysis, and 
other processes supportive of conceptual change). We incorporate both 
these perspectives into our discussion of results. 

Data analysis was conducted using Gunawardena, Lowe, and 
Anderson's (1997) interaction analysis model, which was developed to 
identify and categorize levels of conceptual change and knowledge con- 
struction in the context of a six-day international online debate. This study 
expands Gunawardena et al.'s exploration of knowledge construction and 
conceptual change and applies their model for analyzing online discourse 
to a context different than that used in the original research. In our discus- 
sion of results we address the feasibility and usefulness of applying this 
model to the analysis of conceptual change in a different setting. 

Research Framework 

In considering how best to approach analysis of conceptual change in the 
CMC environment, we (one the course instructor, one a participant in the 
online course) explored the literature on constructivism and conceptual 
change. Although constructivists use many lenses to view the conceptual 
change process, two consistent tenets of the theory hold that learners are 
active builders of their own knowledge and that there is a need for "con- 
necting the new knowledge to be acquired with the existing knowledge 
that students have" (Limon, 2001, p. 358). Researchers have studied con- 
ceptual change in terms of processes that occur in the individual as he or 
she interacts with new ideas (Piaget, 1975; Strike & Posner, 1992); others 
have examined aspects of conceptual change in the context of social inter- 
action (Limon; Vygotsky, 1978). 

Posner, Strike, Hewson, and Gertzog (1982) view conceptual change in 
terms of conditions and effects that occur in the individual. They adopt the 
language of Piaget in defining a two-phase process of change that includes 
assimilation, where students use the concepts they have already mastered 
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to organize and make sense of new phenomena, and accommodation, 
brought on when "current concepts are inadequate.... Then the student 
must replace or reorganize his central concepts" (p. 12). Dole and Sinatra 
(1998) identify factors that affect conceptual change, including personal 
relevance for the individual, adequate background knowledge, sufficient 
cognitive ability, and comprehensible information. They further state that 
change in beliefs can occur on an individual level when students engage in 
a thoughtful, effortful processing of arguments. Limon (2001) cites addi- 
tional factors that contribute to conceptual change, including "motivation, 
learning strategies, epistemological beliefs, attitudes . . . the teacher . . . and 
social factors, such as the role of peers" (p. 365). Such factors can be 
observed in the online setting if participants are encouraged to reflect in 
writing on their own reactions and thought processes and to respond 
thoughtfully to the comments of others. 

Gunawardena et al. (1997) note, "It is important to recognize the inter- 
dependence of both the individual and the social construction of know- 
ledge" (p. 409). Social interaction and the development of a community of 
learners can be influential factors as well, as documented by the growing 
knowledge base on the value of online learning communities (Conrad, 
2002; Palloff & Pratt, 1999; Stacey, 1999). This influence may vary with 
setting; Conrad suggests that the unique characteristics of the online class- 
room, where community is "constructed and maintained as a necessary 
tool for the completion of tasks" (p. 11), may produce a more pragmatic, 
functional version of community than in other online scenarios. What, 
then, are pragmatic and functional ways instruction can be designed to 
promote conceptual change in the online classroom? 

Limon (2001) groups instructional strategies that promote conceptual 
change into three categories: developing cognitive conflict, applying 
analogies, and facilitating "cooperative and shared learning to promote 
collective discussion of ideas" (p. 358). These cognitive tools can be iden- 
tified in online discussion. Another means for assessing the process of 
conceptual change in the online environment is observing the existence of 
cognitive presence in discourse. Garrison et al. (2001) define cognitive 
presence as "the extent to which learners are able to construct and confirm 
meaning through sustained reflection and discourse in a critical com- 
munity of inquiry" (p. 11) and note that "cognitive presence focuses on 
higher-order thinking processes as opposed to specific individual learning 
outcomes" (p. 12). 

The synthesis of these perspectives guided our approach to this re- 
search. Data for the study were gathered from an online course that was 
designed to encourage shared learning and facilitate collective discussion. 
The course units and discussions chosen for analysis were those most 
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likely to motivate students to reflect on and apply their own knowledge 
and experience. The content analysis instrument we chose embeds refer- 
ences to cognitive conflict, analogous thinking, reflection, higher-order 
cognitive processes, and conceptual change in defining its five phases of 
knowledge construction. Beyond content analysis, we hoped that our 
personal involvement in the course, coupled with a qualitative review of 
both individual messages and sequences of messages, might bring to light 
more subtle evidence of conceptual change. 

Method 

Setting 

The online course in this study was sponsored by the Center for Learning 
and Teaching in the West (CLTW), a consortium of K-16 institutions in 
Montana, northern Colorado, and Portland, Oregon, funded by the Na- 
tional Science Foundation's Mathematics and Science Partnership pro- 
gram. One aspect of the partnership's mission is to produce future leaders 
in higher education through a doctoral fellowship program in mathe- 
matics and science education. The core curriculum of this program is 
derived from a set of 15 online course offerings grouped thematically into 
five triads. Each triad is sponsored by one of the five campuses involved in 
CLTW using Web-based course software such as WebCT® or Black- 
board®. Doctoral fellows at any participating campus enroll in these 
courses as they continue their regular research, professional development, 
and teaching activities. This study examines one of the CLTW online 
courses sponsored by Montana State University in the Curriculum- Assess- 
ment-Evaluation triad. The first course in the series, "Curriculum Design: 
The Case of Mathematics and Science," was first offered in fall 2002. The 
course analyzed in this study, "Models of Assessment in Mathematics and 
Science," followed in spring 2003. 

As is typical in CLTW distance learning, the Assessment course in this 
study relied heavily on facilitated participant discussion. The course was 
divided into units lasting from two to three weeks and included comple- 
tion of readings and activities, both scaffolded and unstructured discus- 
sions, and sharing of research in online forums. A typical unit discussion 
generated 150 to 200 discussion messages from the 15 participants. In total 
the various discussions in the Assessment course generated over 2,000 
discussion messages during the spring 2003 semester. 

Collins and Berge (1996) suggest that online instructional roles fall into 
four categories: pedagogical, managerial, social, and technical. The course 
instructor fulfilled all these responsibilities in designing course content, 
building the course shell, introducing students to the software platform 
and to each other, and initiating the course. She also laid the foundation 
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for each subsequent unit by providing an overview of the unit topic; 
laying out guidelines for conducting activities and summarizing the as- 
signed readings (book chapters, journal articles, and online documents); 
and introducing questions to begin each discussion. The instructor, a 
university faculty member with several semesters' experience in designing 
and instructing online courses for graduate students, sought to create an 
environment where "members question one another, demand reasons for 
beliefs, and point out consequences of each other's ideas-thus creating a 
self-judging community" (Garrison et al., 2001, p. 12). 

New discussions were opened for each unit. In the first two units, 
students were assigned to one of three discussion subgroups. Based on 
students' requests for larger groups, subsequent units included two dis- 
cussion groups of seven or eight, each led by a student facilitator. Discus- 
sion groups were shuffled after every two units, and new group facili- 
tators were assigned for each unit. The facilitators, most of them 
experienced educators, were given individual guidance in initiating, 
stimulating, monitoring, and summarizing the discussion for their par- 
ticular unit. The instructor provided each facilitator with what Palloff and 
Pratt (1999) call "expansive questions" designed to promote "deep ex- 
ploration of a topic and the development of critical thinking skills" (p. 
199). 

Stacey and Rice (2002) suggest that although it is the instructor's role to 
establish a secure environment for interaction, that environment can even- 
tually be structured by the instructor but led by students. The instructor 
did have a presence in the unit discussions to comment, answer specific 
questions, and synthesize ideas; however, the student facilitators were 
also expected to develop these skills as part of their fellowship training. 
Besides modeling desirable facilitator traits, the instructor communicated 
with group leaders through individual e-mail, providing appropriate 
questions, suggesting lines of inquiry to pursue, and offering ideas to 
direct and redirect the discussion. In short, the instructor attempted to 
build in the group leaders the skills of facilitation and questioning needed 
to generate thoughtful, quality discussion. 

At the end of each unit discussion, the facilitators completed a survey 
summarizing the key points raised as well as critiquing the quality of the 
discussion. In addition, discussion members were required to complete a 
self-evaluation of their level of participation at the end of each unit discus- 
sion. The four-point rubric asked students to reflect on their efforts to 
interact, if not on the depth of those interactions: 

Did I respond at least once to the questions/issues raised by my 

discussion leader? (1.0) 

Did I submit a second response raising an original question or exploring 
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an issue related to the readings and/ or assignments for the week? (1.0) 

Did I submit a response to at least two other members' messages? (1.0)? 

Did I make a connection between course material and my 
classroom /school? (.5) 

Did I get involved early in the first week, so others could respond to my 
messages? (.5) 

When specific discussions were set up to support a unit activity, students 
were given guidelines both for presenting their activities and for respond- 
ing to the presentations of others. These guidelines focused more on 
stimulating a give-and-take forum atmosphere than on deliberately coach- 
ing higher-level thinking. 

Luebeck, a mathematics education faculty member, was the instructor 
for the assessment course; Bice, at the time a high school science teacher 
and CLTW doctoral fellow, participated as a student in the course. The 
Assessment course was designed and implemented according to CLTW 
standards and in collaboration with the designers of the curriculum and 
evaluation courses, with no research intent influencing any part of the 
process. Only after the course was completed, when we were studying the 
work of Gunawardena et al. (1997) and others regarding knowledge con- 
struction and conceptual change in a structured CMC environment, was 
the present study conceived and designed. We were intrigued by the 
notion of extending the investigations and findings of Gunawardena et al. 
to a new context: that of more collaborative interaction among relatively 
like-minded educators in an online graduate course. 

Description of Participants 

The 15 students in the Assessment course represented a wide variety of 
experiences and education-related professions, including secondary 
teachers, college instructors, professional developers, and doctoral stu- 
dents in content fields. Their familiarity with WebCT®) varied, but all had 
some degree of facility with distance learning and Web-based discussions. 
Twelve of the students had participated in the online CLTW curriculum 
course during the previous semester, which greatly leveled the playing 
field in terms of online experience. Nearly half the students were from 
Montana; others were located in Oregon, Arizona, New Mexico, Alaska, 
South Dakota, and Washington, DC. They worked in both urban and rural 
settings; some dealt directly with underserved populations (primarily Na- 
tive American). 

Selection of Instrument 

To search for evidence of conceptual change in online discussion, we chose 
the method of content analysis defined as "a research technique for the 
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objective, systematic, and quantitative description of the manifest content 
of communication" (Berelson, 1952, cited in Borg & Gall, 1989, p. 519). 
Relatively new to the field, we reviewed the literature on evaluation 
methodologies for computer conferencing and examined several existing 
scales and inventories. We chose the interaction analysis model developed 
by Gunawardena, Lowe, and Anderson (1997, referred to below as GLA) 
based on several criteria. First, the model, with its identification of five 
phases of knowledge construction (sharing ideas, encountering dis- 
sonance, negotiating meaning, and eventually reaching conceptual change 
through processes of synthesis and application), was well suited to content 
analysis. Second, the model was developed to measure quality rather than 
quantity of interaction to determine whether "individual participants 
change their understanding or create new personal constructions of know- 
ledge" (Gunawardena et al., 1997, p. 399) through group interaction. 
Third, we believed that of the instruments we reviewed, GLA was most 
capable of detecting cognitive conflict, analogous thinking, reflection, and 
higher-order cognitive processes: aspects of the learning process that the 
literature suggests are supportive of conceptual change. Finally, the model 
suited our context of discussions among experienced educators. Garrison 
et al. (2001), who examined GLA in developing their own model of critical 
inquiry, note that such models are "more appropriate where applied 
knowledge is valued-particularly adult, continuing, and higher educa- 
tion" (p. 21). Key descriptors for the five levels identified in the GLA 
interaction analysis model are provided in Table 1. 

Data Analysis 

Due to the volume of messages generated in the Assessment course (over 
2,000 total), we felt it necessary to limit the data to be analyzed. Kelly and 
Green (1998) differentiate between arguing a position and changing one's 
conceptions. With this in mind, we chose the three instructional units most 
likely to push students beyond simply sharing their findings or comparing 
(and retaining) their views: units that encouraged students to think deeply 
and invited varied interpretations of the subject matter. These include: 

• Unit 2: Formative Assessment — Students explored 
standards-based approaches to formative assessment in 
mathematics and science, contrasted the purposes and applications 
of formative and summative assessment, and debated the 
usefulness of assessment as a learning tool. 

• Unit 5: Equity and Quality — Students identified and debated issues 
of equity in relation to assessment, considered validity and 
reliability in standardized testing, and contrasted state- and 
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Table 1 

Summary of GLA Interaction Analysis Model (Adapted from Gunawardena et al., 1997) 


Level 

Identity 

Description 

1 

Sharing and comparing 

Statement of observation/opinion 
Statement of agreement 
Corroborating examples 
Asking/answering questions 
Identification of a problem 

2 

Discovery and exploration of dissonance or 
inconsistency 

Identifying/stating areas of disagreement or 
inconsistency 

Asking/answering clarifying questions 
Restating a position/advancing an argument 

3 

Negotiation of meaning/co-construction of 
knowledge 

Negotiation/clarification of terms 
Identifying areas of agreement/overlap 
Proposal/negotiation of compromise 
Proposal of metaphors/analogies 

4 

Testing and modification of the proposed 
synthesis 

Testing proposed synthesis 

Testing against existing cognitive 

schema/experience/contradictoryevidence 

5 

Agreement and application of new meaning 

Summarization 

Applications of new knowledge 
Metacognitive statements illustrating 
self-knowledge of conceptual change 


national-level tests with classroom assessment in terms of 
appropriate and useful reporting. 

• Unit 7: Assessment Utopia — Students investigated the alignment 
of local, state, and national assessments with standards for 
mathematics and science at similar levels, debated the potential 
roles and responsibilities of "stakeholders" (e.g., students, teachers, 
parents, policymakers), and investigated alternatives to traditional 
high-stakes testing. 

The remaining units generated excellent debate-style discussions that 
challenged the thinking of participants, but offered less evidence of con- 
vergent thinking. Unit 1 was more didactic in tone, used to develop a 
common vocabulary in terms of mathematics and science standards, types 
of assessment strategies, and familiar assessment issues. Unit 3 examined 
the range and scope of external assessment practices designed for local, 
state, and national purposes and introduced students to commonly held 
positive and negative views and the position statements of national educa- 
tion organizations. In Unit 4 students conducted individual research on an 
assessment topic of personal interest and presented their findings in an 
online forum. Unit 6, again didactic, revisited the contrast between forma- 
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tive and summative assessment data and described data-driven school 
change models. Several other discussions (the No Child Left Behind Educa- 
tion Act, how home schooling affects high-stakes assessment) arose spon- 
taneously and were pursued with vigor, but not subject to the same 
instructional design. 

We examined complete transcripts from each of the three selected 
instructional units and coded all messages individually (a total of 484 
messages posted by the 15 students and instructor). A full message was 
determined as the unit of analysis. Too often single sentences or phrases 
taken out of context must be scored at the lowest level of "sharing and 
comparing" information, even though the message as a whole represents 
a higher level of knowledge construction. Garrison et al. (2001) further 
note that submessage units are more difficult to identify between coders 
and can reduce reliability. 

Before coding the research material, we practiced coding by applying 
the GLA model to archived discussions from the online CLTW curriculum 
course, as well as to transcripts of interactions among teachers in an online 
mentoring program. Each individual coder was free to assign separate 
scores to distinct submessages in a single posting, but reported only one 
summary code for each posting when comparing results. We adopted 
Garrison et al.'s (2001) method of arriving at a meaningful final code for 
each message: "Code down (i.e., to the earlier phase), if it is not clear which 
phase is reflected; and code up (i.e., to the later phase), if clear evidence of 
multiple phases is present" (p. 9). To test for interrater reliability, we first 
individually coded all messages from a subset of five students (w=220), 
achieving a reliability rating of Kappa=.52. We discussed differences in 
coding for individual messages until a final code was agreed on for all 
messages. Preserving the codes for the five initial subjects, we then 
returned to the data to code individually the remaining messages. A 
reliability value for all 484 messages was calculated at Kappa=.83. As 
before, disagreements in coding were resolved through mutual analysis: 
these were typically due to oversight and were quickly resolved on 
reviewing the message. 

Although discussion transcripts provided by computer-mediated com- 
munication are an invaluable resource, we are compelled to point out 
some of the limitations of content analysis in this context. In the 
asynchronous environment, participants have time to process their reac- 
tions and reflect on new ideas before responding to prompts. Garrison et 
al. (2001) suggest that "Observers view only that subset of cognitive 
presence that the participants choose to make visible in the conference" (p. 
13). Members of an online discussion may put forward only the final 
results of a complex cognitive process, leaving intermediate steps unex- 
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amined. Indeed, we may not even see final results if students keep their 
cognitive processes internal or reserve a summary of their learning for 
individual assignments (in the Assessment course these included reading 
syntheses, a research report, and a final state assessment project). How- 
ever, these phenomena are reflective of studying cognitive activity in any 
setting. Even if one can only observe snapshots of the conceptual change 
process, the online record serves as a remarkable source of evidence. 

Results and Discussion 
Evidence of Conceptual Change 

A total of 484 messages were coded using the interaction analysis model 
developed by Gunawardena et al. (1997). The total numbers of messages 
coded at each GLA phase are shown in Table 2. The findings indicate clear 
evidence of cognitive conflict and analogous thinking as identified in GLA 
Phases 2 and 3. Far fewer indicators of reflection, metacognitive activity, 
and higher-order cognitive processes were evident: these aspects of the 
conceptual change process generally occur in Phases 3-5. Only seven of 
484 (roughly 1.5%) of the messages represented testing and modification 
of proposed synthesis (Phase 4) and/ or agreement and application of new 
meaning (Phase 5). In addition, 43 messages (roughly 9%) revealed that 
participants were engaged in negotiating meaning and constructing 
knowledge (Phase 3). 

Significance of interaction 

Although the quantitative results provide limited evidence of conceptual 
change in individual messages, from a qualitative perspective the tran- 
scripts reveal numerous instances in which interaction among learners 
precipitated negotiation of new meaning. Additional evidence of concep- 
tual growth and change is apparent when messages are read in the context 
of their respective discussion threads. With its emphasis on disagreement 
and negotiation, the GLA instrument tends to overlook the cumulative 
effects of an exchange of multiple viewpoints and the sharing of ideas that 


Table 2 

Coding Results for Three Units in the CLTW Assessment Course 


GLA Phase 

1 

2 

3 

4 

5 


Sharing/ 

Dissonance/ 

Negotiation/ 

Testing/ 

Agreement/ 


Comparing 

Inconsistency 

Construction 

Modification 

Application 

Messages coded 

at each phase 

290 

144 

43 

6 

1 
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build on each other rather than conflicting. Evidence of testing, modifica- 
tion, and synthesis by the group may not be recognizable in the coding of 
individual messages, but is present in the interactive whole that spans a 
complete discussion thread. 

As an example, consider how the following discussion thread inter- 
weaves personal experience, reaction, and reflection on the path to con- 
ceptual change. In this series of excerpted statements, participants are 
discussing how a transitory student population affects assessment prac- 
tices and outcomes. At the conclusion of this exchange about educating 
transient students. Student P offers a metacognitive statement about her 
changed way of thinking. 

Student R (Facilitator): How do we assess these kids that move in and out 
of our schools like a morning fog that burns off by noon? Is testing more 
often the answer? 

Student D: I have a suggestion for at least part of the problem. . . . 
Standards-based report cards in each subject area would provide a 
comprehensive picture. 

Student P: Student D's suggestion would be great assuming content 
standards were consistent across districts and across states . . . directed by 
the national standards . . . Why aren't we? 

Student W: I realize that education stems from local control, and I agree, 
but ... In my other life as a plumber, we don't reinvent the wheel each job. 

We have a basic framework in the plumbing code. 

(Six messages omitted) 

Student M: A checklist of what had been covered up to that point in the 
year might do the new teacher a lot of good. 

Student R (Facilitator): I like this check sheet idea.. . . Where he/ she is in the 
scope and sequence of the curriculum might be a better approach than an 
exit or entrance exam.. . . How we determine this is the challenge. 

Student W: What if kids had to spend a day taking entrance exams 
whenever they moved to a new school . . .? Would they wish to move a 
lot? What if the parents had to spend a day getting a child all set up in a 
new school? Would they wish to move a lot? 

Student P: I don't know that parents "want to move a lot." I really believe 
that in most cases it is a necessity. ... At least that is what I've seen in the 
communities in which I have taught. However, I really like the idea of 
entry testing. ... it would make parents more aware of the impact that 
changing schools might have on their children. 
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Student D: My students tell me plenty of reasons why they switch schools 
and they do it regularly. ... If the student is having a hard time at school, 
scholastically, not being played on the sports team, fights with others, bad 
influences . . . pregnancy, poverty . . . the list goes on and on. . . . Most of 
these moves are within a 100-mile radius, but if there is a willing relative 
farther away they may make that move as well. 

Student W: To me, the child's education and continuity of education is 
important, even if that means parents putting some plans on hold.. . . Most 
of my midyear moves bring an assortment of baggage with them. 

Student P: Student D paints a picture of school switches and changes I 
have not experienced. Given this information, I can imagine that it is and 
would be frustrating to have to deal with such frequent and seemingly 
senseless changes from one school to another. Thanks ... it helps me to 
relate. 

There is considerable evidence of harbingers of conceptual change such 
as cognitive conflict stimulated by the interactive process of negotiating 
contrasting (and often conflicting) perspectives, as in the following excerpt 
where two students exchange views about the nature of standards-based 
instruction: 

Student Facilitator: Standards-based teaching /learning sounds a lot like 
mastery learning. Do you think so? I have had [an awful] experience with 
mastery learning. 

Student D: I'd have to agree with you. Sometimes it might be better to 
leave a subject for a while till your mind is better ready to handle it. 

Student S: I think standards-based is a lot different than mastery learning. 

I do think there needs to be a set of standards that we as a profession 
agree is our obligation to teach. If not, we would be all over the place. . . . 
[National Council of Teachers of Mathematics] math standards are not 
mastery learning at all — they inform teachers about the whats (and even 
some of the hows) to teaching math. 

Student D: To take the alternate point of view, how is becoming proficient 
in fractions any different than mastering them? . . . The only difference is 
we don't require complete mastery before going on to the next topic. 

Finally, use of analogies is evident throughout the three course units as 
in the plumbing analogy given above. Another student clarifies her point 
by likening curriculum standards to standards for industry: 

Student D: Maybe we need to think of [education] as a business. Because 
we are not trying to attract "clients," we often just go our own way. . . . But 



34 


JENNIFER L. LUEBECK and LAWRENCE R. BICE 


most of the industrialized world uses national, even international, 
standards. . . . How awful would it be to buy a car in Montana, then move 
to Oregon, and find that nobody there knew how to work on it! 

Increased Cognitive Activity Over Time 

A comparison of coding results from the three instructional units in the 
Assessment course reveals that overall, participants' higher-level cogni- 
tive activity (as measured by GLA) increased over time (see Figure 1). 
Although we must interpret this result with caution, it suggests that as 
participants became more familiar with course procedures, their fellow 
students, and the subject matter, they were able to interact at higher 
phases in the GLA model. It is also noteworthy that the highest levels of 
cognitive activity (messages coded at Phases 4 and 5) occurred in the Unit 
5 discussion about diversity and equity issues in standardized assess- 
ments. This was an area where students' prior knowledge was particularly 
challenged both by reading scholarly articles and by assimilating the 
strongly held views and personal experiences of others. At times a single 
discussion prompt generated 30 or more responses as arguments were 
advanced and new meanings were negotiated and tested. 

Conceptual Change and Course Characteristics 

We offer three observations specific to this study and the Assessment 
course that speak to the potential of online learning to support and en- 
courage conceptual change. First, our results suggest that the geographic 
and experiential diversity of this group of online learners contributed to 
the development of new knowledge. Collectively the participants began 
the Assessment course with an awareness of educational policy in seven 
states, instructional expertise in mathematics and science grades 8-14, and 



Figure 1 . Percentage of messages coded at Phase 3 or higher across three 
Assessment course units. 
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a wide range of perspectives on assessment based on personal experience. 
Discussions and forum-style sharing sessions were naturally enhanced by 
the multiplicity of examples, vignettes, and comparisons introduced by 
the participants. Two (distinct) examples are given below. 

Example 1: Student W. As I've been in reservation schools over this past 
semester, I can't help but watch the results of students in poverty. It 
certainly overshadows the culture. I watched poverty in inner city 
Harrisburg [PA] and I saw much the same results on students. Poverty 
seems to make people give up, not care, not try, not have hope. 

Example 2: Student D (a teacher of Native American students). The answers I 
get on tests are much more European than the ones I get when I ask a 
question in class or hear two students talking amongst themselves.. . . 

What is so fun to listen to is a conversation in Crow. The language is very 
descriptive, each word describes what they see. 

Second, analysis of these discussions supports the notion that the op- 
portunity for teachers to engage in online learning concurrently with their 
own teaching enhances social interaction and supports conceptual change. 
Berge (2002) makes it clear that situated learning, or learning in context, is 
"critical for determining meaning" (p. 183). Many of the students were 
actively engaged in curriculum planning and assessment activities while 
taking the Assessment course and were able to provide real-time commen- 
tary and comparisons. They found their viewpoints and beliefs challenged 
not only by readings and research, but by peers who were experiencing a 
different reality at that very moment in their state, district, and classroom. 

Finally, this study demonstrates the importance of deliberate "cogni- 
tive coaching" if the desired outcome is higher-level thinking, negotiation 
of meaning, and eventual conceptual change. Vrasidas and Mclsaac (1999) 
found that increased structure led to more dialogue and interaction, but 
our results suggest that structure is not enough to influence higher-level 
thinking. In this online course the audience comprised self-motivated 
educators and scientists, discussion facilitators were provided with 
thought-provoking questions, and students were given guidelines for 
their participation and required to evaluate their own interactions; how- 
ever, significant evidence of conceptual change did not appear. Knowlton 
(2003) warns us that "Critical thinking is abstract, and students may not 
know how to operationalize and apply it in their discussion contributions" 
(p. 35). Despite the behind-the-scenes support they received, graduate 
students in the Assessment course discussions were generally not success- 
ful in elevating the discussion to the highest levels of the GLA model. 
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Adaptability of the Instrument 

The results of this study did not differ substantially from the findings of 
Gunawardena et al. (1997) in terms of Phase 4 and 5 messages. However, 
the results at the opposite end of the scale are significantly different. There 
were far fewer messages at Phase 1 among the 15 students participating in 
the Assessment course (61% at Phase 1) than for the participants in the 
online debate studied by Gunawardena et al. (93% at Phase 1). Participants 
in the Assessment course appeared to move beyond the sharing-and-com- 
paring stage more easily than those involved in the international debate. 
This outcome was not unexpected given that in the global debate context, 
relative strangers were expected to adhere to and defend a specific as- 
signed point of view. By contrast, the Assessment course invited par- 
ticipants with similar backgrounds and some degree of familiarity to 
exchange views over a much longer period. We recognize that develop- 
ment of the GLA model was grounded in a particular context and do not 
suggest that this is sufficient to identify cognitive activity and conceptual 
change in all other online settings. However, the GLA model was useful in 
quantifying the cognitive activity of the online learners in this study. 

Overall, although the interaction analysis model designed by 
Gunawardena et al. (1997) is a useful starting point, we found that the 
debate context used to develop the model severely limits its application to 
other settings. Development of the GLA model assumed — in fact con- 
trived — a contentious context with the goal of resolving opposing view- 
points. An exploration of issues in a cohort of graduate students is not 
likely to produce such a scenario. In this study difficulties arose with 
clearly differentiating Phases 2 and 3 messages because indicators for 
those phases in the GLA model imply the existence of disagreement and a 
need to resolve conflict. In fact some of the low coding scores we report 
may be the result of our hesitancy to assign higher values to messages that 
may have demonstrated movement toward conceptual change, but lacked 
features that matched GLA indicators referring to dissonance, disagree- 
ment, and a need to negotiate opposing views. A model that moves away 
from learning through rational argument ("I will convince you") to a more 
fluid process of learning through interaction ("Let's pursue this together") 
would be more suitable to the graduate student experience. 

Conclusion 

The results of this study suggest that the building blocks of conceptual 
change, and conceptual change itself, can be identified through the ex- 
amination of online discussion among adult learners in a graduate course. 
However, there is limited evidence to indicate that interactive discussion 
in the CLTW Assessment course resulted in conceptual change and new 
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understandings for individual students. This may have been due in part to 
the limited abilities of student facilitators to push students to higher cogni- 
tive levels. We contend that it is also due in part to the inability of the data 
analysis model to interpret adequately knowledge construction in the 
graduate course context. 

Knowlton (2003) asks whether evaluation systems for online discus- 
sion can transcend disciplinary lines. The pedagogical, social, managerial, 
and technical components of online instruction identified by Collins and 
Berge (1996) do not directly address how an instructor delivers specific 
content or how it is received. However, Conrad (2004) found that new 
online instructors were most concerned about successfully delivering con- 
tent. Can an instrument be developed to determine effectively whether 
conceptual change occurs in the realm of teaching and learning mathe- 
matics and science? 

White (2002) observes that conceptual change may be viewed as dis- 
cipline- or even topic-specific. The importance of discourse and communi- 
cation in supporting the learning of mathematics and science is well docu- 
mented by the national standards for both disciplines and in related 
research (National Council of Teachers of Mathematics, 2000; National 
Science Teachers' Association, 1996). Our next steps are to add to this work 
by investigating how the online arena contributes to teaching and learning 
science and mathematics. In particular, we are encouraged to pursue the 
development of a content analysis model that specifically attends to the 
construction of mathematics and science content knowledge. 
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